Systems and methods for panoramic imaging

ABSTRACT

Systems and methods for panoramic imaging are disclosed herein. An exemplary system of the disclosed subject matter for panoramic imaging includes a fisheye lens, a mirror optically coupled to the fisheye lens, and a detector for capturing light incident on the fisheye lens and reflected from the mirror. In some embodiments, the mirror is a spherical mirror.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/340,466, entitled “Systems And Methods For Panoramic Imaging”, filedJul. 24, 2014, which is a continuation of U.S. patent application Ser.No. 13/092,591, entitled “Systems And Methods For Panoramic Imaging”,filed Apr. 22, 2011, which issued as U.S. Pat. No. 8,817,066 on Aug. 26,2014; which is a continuation of U.S. patent application Ser. No.12/726,029, entitled “Systems And Methods For Panoramic Imaging”, filedMar. 17, 2010, which issued as U.S. Pat. No. 8,767,037 on Jul. 1, 2014;which is a continuation of International Application PCT/US2008/077258,entitled “Systems And Methods For Panoramic Imaging”, filed on Sep. 22,2008, which claims priority to U.S. Provisional Application No.60/974,338 entitled “Cara-Fisheye Camera for Panoramic Imaging”, filedon Sep. 21, 2007, each of which is incorporated herein by reference inits entirety and from each of which priority is claimed.

STATEMENT REGARDING FEDERALLY-SPONSORED RESEARCH

This invention was made with government support under N00014-05-1-0032awarded by the Office of Naval Research. The government has certainrights in the invention.

BACKGROUND

1. Field

The present application relates to systems and methods for panoramicimaging.

2. Background Art

Wide-angle and panoramic imaging have had significant impact on avariety of real-world applications. For example, being able to “see” inall directions provides situational awareness in surveillance andautonomous navigation tasks. Image-based immersive virtual realityenables realistic exploration of indoor and outdoor environments.Panoramic video conference and telepresence devices aid in effectivevirtual collaboration. All these applications rely on capturinghigh-quality panoramic videos. In many of these applications, it is notrequired to capture the entire spherical field of view (FOV). It issufficient to capture 360 degree panoramas with a reasonable verticalfield of view that has more or less equal coverage above and beneath thehorizon. For instance, in the case of video conferencing, it is onlynecessary to capture the sitting participants and the table. The same istrue for ground navigation, where the vehicle only needs to view theroads and the obstacles around it.

Current techniques for capturing panoramic videos can be classified aseither dioptric (systems that use only refractive optics) orcatadioptric (systems that use both reflective and refractive optics).Dioptric systems include camera-clusters and wide-angle or fisheye lensbased systems. Catadioptric systems include ones that use single ormultiple cameras with one or more reflective surfaces.

A popular dioptric solution is the use of fisheye lenses. Since fisheyelenses have a hemispherical FOV, fisheye lenses can only be used tocapture hemispherical panoramas. D. Slater, “Panoramic Photography withFisheye Lenses”, International Association of Panoramic Photographers,1996, proposed the use of two back-to-back fisheye lens cameras toobtain full spherical panoramas. Similar to camera-cluster approaches,this approach suffers from large parallax. In addition, if the goal isto capture a reasonable vertical coverage about the horizon then interms of optics, image detection and bandwidth, it is wasteful tocapture the entire sphere.

Catadioptric system employ one or more reflective surfaces capable ofprojecting a wide view of the world onto a single sensor. Several ofthese designs satisfy the so-called single viewpoint (SVP) constraint(e.g., that the entire image appears to be capture from a single pointof reference). S. Baker and S. Nayar, “A Theory of Single-ViewpointCatadioptric Image Formation”, IJCV, 35(2):175-196, 1999, derived afamily of mirrors that satisfy the SVP constraint, when used withperspective lenses. Since in these designs perspective cameras view theworld through highly curved reflective surfaces, the vertical resolution(number of pixels per unit elevation angle) of the computed panoramas isnot uniform and is, in general, poor near the apex of the mirror.

Yagi and Kawato, “Panorama Scene Analysis with Conic Projection,” IEEEInternational Workshop on Intelligent Robots and Systems, Vol. 1, pp.181-187, 1990, and Lin and Bajsey, “True Single View Point Cone MirrorOmni-Directional Catadioptric System, ICCV, pp. 102-107, 2001, have usedconical mirrors in their catadioptric designs, with the lattersatisfying the SVP constraint. In these examples, a better verticalresolution is obtained. The conical mirror acts as a planar mirror inthe radial direction and hence the vertical resolution is the same asthat of the camera. However, this restricts the maximum vertical FOV ofthe panorama to half the FOV of the camera. For instance, if the goalwas to capture a panorama with a 60 degree vertical field of view, thecamera must have a 120 degree field of view. Even if such a system isused, the resolution of the panorama approaches zero at the apex of thecone. More importantly, conical mirrors are known to produce strongoptical aberrations due to astigmatism that fundamentally limit theresolution of the captured panorama. Thus, there is a need for improvedsystems and methods for capturing panoramic images.

SUMMARY

Systems and methods for capturing a panoramic image of a field of vieware disclosed herein. An exemplary system of the disclosed subjectmatter for capturing a panoramic image of a field of view includes afisheye lens, a mirror optically coupled to the fisheye lens forreflecting an image of the field of view through the fisheye lens, and adetector, optically coupled to the fisheye lens and the mirror, forcapturing the panoramic image. In some embodiments, the mirror is aspherical mirror and can be positioned coaxially with regard to thefisheye lens. In some embodiments, the system is further configured tosatisfy the following conditions that the upper angle of a desired fieldof view is equal to the upper angle of a field of view of the fisheyelens, the lower angle of the desired field of view is less than or equalto the lower angle of a field of view of the mirror, and the angle ofoverlap between the field of view of the mirror and the field of view ofthe fisheye lens is equal to the difference between the lower angle ofthe field of view of the fisheye lens and the upper angle of the fieldof view of the mirror.

The system for panoramic imaging can further include a processor and amemory storing program instructions to cause the processor utilize thedetector to capture a first portion of the image of the field of viewtransmitted directly through the fisheye lens, and capture a secondportion of the image of the field of view reflected from the mirrorthrough the fisheye lens. In some embodiments, the memory stores programinstructions that when executed by the processor, further cause theprocessor to stitch together the first portion of the captured image andthe second portion of the captured image to form a panoramic image. Insome embodiments, the stitching together of the first portion of thecaptured image and the second portion of the captured image is performedutilizing a scale-invariant feature transform algorithm.

In some embodiments, the memory stores program instructions that furthercause the processor to calibrate the system. The memory can also storeprogram instructions that further cause the processor generate a tableof correspondence between captured image pixel locations and panoramapixel locations. In some embodiments, the memory stores programinstructions that further cause the processor utilize the detector tocapture the first portion of the image and capture the second portion ofthe image simultaneously.

An exemplary method of the disclosed subject matter for capturing apanoramic image of a field of view includes positioning a mirror a firstdistance from a fisheye lens, utilizing the fisheye lens to capture afirst portion of the image of the field of view transmitted directlythrough the fisheye lens, utilizing the fisheye lens to capture a secondportion of the image of the field of view reflected from the mirrorthrough the fisheye lens, and stitching together the first and secondimages to compose a panoramic image. The method can further includelocating image pixels correspondingly found in both the first and secondportions of the captured image, determining calibration settings for themirror, determining calibration settings for the fisheye lens, andgenerating a table of correspondence between a pixel location on thecaptured image and a pixel location on the panoramic image.

In some embodiments, the capture of the first and second portions of theimage are performed further utilizing a detector. The capture of thefirst and second portions of the image can be performed simultaneously,and stitching together the first and second portions of the image can beperformed utilizing a scale-invariant feature transform algorithm. Thedetermination of calibration setting for the mirror can be performed bydetermining the center of curvature and the orientation of the mirror.The determination of calibration setting for the fisheye lens can beperformed by determining the projection function of the fisheye lens.

In some embodiments, the generation of a table of correspondence betweena pixel location on the captured image and a pixel location on thepanoramic image further includes determining an image location of thepixels, calculating a corresponding panorama location for the pixelslocated in the first portion of the captured image, calculating acorresponding panorama location for the pixels located in the secondportion of the captured image, and calculating a corresponding panoramalocation for the pixels located in both the first and second portions ofthe image. The calculation of the corresponding panorama location forthe pixels located in both the first and second portions of the capturedimage can further include using a smoothing technique to calculate thecorresponding panorama location for the pixels located in both the firstand second portions of the captured image. The stitching together thefirst and second portions of the captured image to compose a panoramicimage can further include using the correspondence table to map thefirst and second portions of the captured image into a panoramic image.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated and constitute part ofthis disclosure, illustrate embodiments of the disclosed subject matter.

FIG. 1 is a diagram of a system for panoramic imaging in accordance withan embodiment of the disclosed subject matter.

FIG. 2(a) illustrates of a field of view captured by a detector inaccordance with an embodiment of the disclosed subject matter.

FIG. 2(b) illustrates an panoramic view of a field of view captured by adetector in accordance with an embodiment of the disclosed subjectmatter.

FIG. 3 is an exemplary design of a system for panoramic imaging inaccordance with an embodiment of the disclosed subject matter.

FIG. 4(a) is an image of an exemplary system for panoramic imaging inaccordance with an embodiment of the disclosed subject matter.

FIG. 4(b) is an image of an exemplary mirror assembly for use with apanoramic imaging system in accordance with an embodiment of thedisclosed subject matter.

FIG. 5 is an exemplary chart of a method for panoramic imaging inaccordance with an embodiment of the disclosed subject matter.

FIG. 6 is a combined image of a mirror-panoramic and fisheye-panoramicof an overlap area illustrating overlap features in accordance with anembodiment of the disclosed subject matter.

DETAILED DESCRIPTION

The systems and methods described herein are useful for panoramicimaging both above and below the horizon. The disclosed subject matterprovides a simple and compact system utilizing a fisheye lens and amirror to capture overlapping panoramic images spanning the horizon.Those images are then stitched together to form a single, highresolution panoramic image of the selected areas of greatest interest tothe user.

FIG. 1 illustrates an exemplary design of a system 100 for panoramicimaging in accordance with the disclosed subject matter. The system 100includes a fisheye lens 110 optically coupled to a mirror 112. Adetector 114, e.g., a 7 frame/sec Lumnera Lw570 digital video camerawith a 5 megapixel sensor, can be utilized to capture an image of lightrays 116 directly incident upon the fisheye lens 110 and light rays 118reflected from mirror 112 through fisheye lens 110. All of the anglesrepresented in FIG. 1 are measured from the vertical center line L,unless indicated otherwise.

Mirror 112 has the characteristics of a diameter d_(m), e.g., 50.26 mm,and a radius r_(m), e.g., 70.12 mm. The mirror 112 can be positioned ata distance h_(m), e.g., 2.6 mm. Fisheye lens 110 has the characteristicsof a vertical field of view of Ω_(ƒ), e.g., 97.5° (the field of view ofa fisheye lens is usually twice its vertical coverage, e.g., a 180°fisheye lens covers 90° in the vertical direction). The fisheye lens 110can be any commercially available fisheye lens, e.g., a FujinonFE185C046HA-1 fisheye lens with a manufacturers listed field of view of185° (with a measured field of view of 195°), or alternatively it can bea custom made fisheye lens.

FIG. 1 further illustrates that fisheye lens 110 can have a field ofview extending from the fisheye lower limit θ_(ƒl) to the fisheye upperlimit θ_(ƒu). Mirror 112 can have a field of view extending from themirror lower limit θ_(ml) to the mirror upper limit θ_(mu). The system100 can have an overall field of view extending from the lower limitθ_(al), e.g., 120°, to the upper limit θ_(au), e.g., 65°, for a totalpanoramic field of view Δθ_(a)=(θ_(al)−θ_(au)), e.g., 55°, whereθ_(au)=θ_(ƒu) and θ_(al)≦θ_(ml) in an exemplary embodiment. The system100 can have an overlap Δθ_(o) in the fields of view of the fisheye lens110 and the mirror 112, such that Δθ_(o)=(θ_(ƒl)−θ_(mu)).

In one exemplary embodiment, an overlap Δθ_(o) of 20-30 pixels can besufficient for blending. Alternatively, if the size of detector is muchsmall than the overlap Δθ_(o) can be 20°. The system 100 is thus definedby the parameters:

θ_(au)=θ_(ƒu)  (1a)

θ_(ml)≧θ_(al)  (1b)

Δθ_(o)=(θ_(ƒl)−θ_(mu))  (1c)

Furthermore, the lower field of view angle for the fisheye lens, θ_(ƒl),satisfies the requirement that θ_(ƒl)=Ω_(ƒ)/2. Parameters (1a) and (1b)state that the mirror and the lens should be positioned such thatneither obstruct the panoramic field of view, and parameter (1c) ensuresthat no part of the desired panoramic field of view is omitted from thecaptured image. Thus, parameters (1a)-(1c) can be set such that thedesired panoramic field of view is captured by the detector 114.

FIG. 2(a) illustrates an overall field of view, as captured by detector114, for an exemplary embodiment of the disclosed subject matter. AreaA_(ƒ) represents the image captured from light rays directly incidentupon fisheye lens 110, while area A_(m) represents the image capturedfrom light rays reflected from mirror 112. As shown in FIG. 2(a), areaA_(ƒ) has an angular width of Δθ_(ƒ)=(θ_(ƒu)−θ_(ƒl)) and area A_(m) hasan angular width of Δθ_(m)=(θ_(mu)−θ_(ml)). Areas A_(o) represents theoverlap of areas A_(ƒ) and A_(m), and have an angular widths ofΔθ_(o)=(θ_(ƒl)−θ_(mu)). Though FIG. 2(a) shows two separate areas A_(o),those skilled in the art will appreciated that these two areas in factdepict the same image. FIG. 2(b) illustrates an unwrapped panorama viewof the field of view represented in FIG. 2(a).

FIG. 3 shows an exemplary panoramic imaging system designed inaccordance with the disclosed subject matter. The cata-fisheye conceptallows for a camera designs with a wide range of fields of view byvarying the parameters (e.g., shape and position) of mirror 112 and thefield of view of fisheye lens 110. In accordance with the disclosedsubject matter, mirror 112 can have a wide range of shapes, e.g.,spherical, paraboloidal, hyperboloidal, conical, or even a complexaspherical shape. In one exemplary embodiment, mirror 112 is a sphericalmirror. In the same or another embodiment, mirror 112 is coaxial withfisheye lens 110.

FIG. 4(a) illustrates an exemplary embodiment of system 100. Asillustrated in FIG. 4(a), the total size of the system can be 10 cm highby 7 cm wide by 5.5 cm deep. As shown in FIGS. 4(a)-(b), mirror 112 ispart of a mirror assembly 410. In one embodiment, mirror assembly 410can be a detachable unit, as shown in FIG. 4(b), which can be secured tothe system 100 by various attachment means, e.g., by screw threads. Themirror assembly 410 can be composed of a top and a cylindricaltransparent wall which can have screw threads on the attaching edge, forsecuring mirror assembly 410 to panoramic imaging system 100. Exemplarysystem 100 is further connected to a processing unit (not shown), e.g.,a standard desktop computer, by a connector 412 that is compatible withdetector 114, e.g., a standard USB cable.

Another design parameter that can be set is the amount of overlapbetween the upper and lower views. Since the stitching of the two viewstakes place in the panoramic image, the overlap can be specified interms of a number of pixels, p_(o). The corresponding angular overlapfor the two fields of view is expressed by:

Δθ_(o)=Ω_(ƒ)/2−ƒ⁻¹[ƒ(Ω_(ƒ)/2)−p _(o)].  (2)

Here, r=ƒ(θ) is the projection function of the fisheye lens, where r isthe radial distance of a pixel from the image center and θ is theelevation angle of the incident ray with respect to the optical axis ofthe fisheye lens.

In one exemplary design, fisheye lens 110 is assumed to have a singleviewpoint positioned at origin O, as shown in FIG. 3. Though at presentfisheye lenses do not exactly have a single viewpoint, any problemsarising as a result of this assumption can be solved by addingtolerances to the limits of the desired field of view. Continuing withFIG. 3, h_(l) is the distance between O and the tip of the fisheye lens110 and d_(l) is the width of the fisheye lens 110 including its casing.In an exemplary embodiment, spherical mirror 112 has a radius ofcurvature r_(m) and width d_(m) which are to be determined, and ispositioned at a chosen distance h_(m) from the lens.

As shown in FIG. 3, P is a point on the rim of mirror 112. In oneexemplary embodiment and according to the parameters 1(a)-(c), set forthabove, the position of P and the surface normal {circumflex over (n)} atP (and hence the radius and center of curvature of mirror 112) are suchthat an incoming ray 120, at angle θ_(mn), is reflected by P towards O.This condition is satisfied if the directional angleθ_({circumflex over (n)}) at P satisfies:

θ_({circumflex over (n)})=½(π+θƒu+θ _(mu)).  (3)

From the above, the radius of curvature and the width of the mirror aredetermined by:

$\begin{matrix}{{r_{m} = \frac{\left( {h_{m} + h_{l}} \right)\sin \mspace{11mu} \theta_{fu}}{{\sin \left( {\theta_{fu} - \theta_{\hat{n}}} \right)} - {\sin \mspace{11mu} \theta_{fu}}}},} & (4)\end{matrix}$

$\begin{matrix}{d_{m} = {\frac{2\left( {h_{m} + h_{l}} \right)\sin \mspace{11mu} \theta_{\hat{n}}\mspace{11mu} \sin \mspace{11mu} \theta_{fu}}{{\sin \left( {\theta_{fu} - \theta_{\hat{n}}} \right)} - {\sin \mspace{11mu} \theta_{fu}}}.}} & (5)\end{matrix}$

In this exemplary embodiment, the radius of curvature r_(m) and thewidth d_(m) are computed as a function of the position h_(m) of mirror112 with respect to fisheye lens 110. While any arbitrary position h_(m)and a mirror 112 with corresponding radius of curvature r_(m) and widthd_(m) can satisfy the upper limit parameter (1a) and the overlapparameter (1c), that does not necessarily ensure that the fisheye lens110 will not obstruct the desired panoramic field of view (e.g.,parameter (1b)). To determine the smallest h_(m) for which θ_(ml)≧θ_(al)a linear search can be used, for example, by using a well knowniterative method utilizing the above equations.

A person skilled in the art will appreciate that there are severaldesign solutions for a desired panoramic field of view based upon thefield of view of fisheye lens 110 and the desired overlap Δθ_(o) betweenthe upper and lower views. Table 1 illustrates exemplary solutions forfive different fisheye lens fields of view:

TABLE 1 Panorama FOV Fisheye FOV Design Solution (in mm) (Required)(Chosen) r_(m) d_(m) h_(m) +30° to −30° 170° 70.00 47.88 4.6 (60° to120°) 180° 76.36 44.45 4.5 190° 81.49 42.18 4.4 200° 96.22 39.73 4.2210° 106.98 37.15 4.1

FIG. 5 illustrates an exemplary method 500 for panoramic imaging inaccordance with the disclosed subject matter. In one embodiment, mirror112 is position 510 and an image is captured 520. The portion of theimage comprising the light directly incident upon the fisheye lens 110can be captured 522 separately from the capture 524 of the portion ofthe image comprising the light reflected from mirror 112 or bothportions can be captured 520 simultaneously. The portions are captured520 utilizing the detector 114. Upon capturing 520 the image, in anexemplary embodiment, the location of a number of pixels located in boththe fisheye and mirror images are determined 530. This determination 530is made, for example, by utilizing a feature specific algorithm, e.g., ascale-invariant feature transform (SIFT) algorithm.

In one exemplary embodiment, once the design parameters for the imagingsystem 100 have been selected the system 100 is calibrated 540. In orderto geometrically map the captured image to the upper and lower panoramasand then stitch them, it is useful to known the relationship between thelight rays entering the camera and the pixels in the captured image.This computation is governed by the design parameters; however, theactual position and orientation of the mirror may not adhere strictly tothe design specifications, as a result of manufacturing tolerances.Another factor is the possible inaccuracy in the assumed positionfisheye lens viewpoint O. As a result, calibration 540 of the system 100is performed by calibrating 542 fisheye lens 110 and by calibrating 544mirror 112. In one exemplary embodiment, calibration 542 of fisheye lens110 is performed by determining the fisheye projection function r=ƒ(θ),and calibration 544 of mirror 112 is performed by determining the actualcenter of curvature (x_(m), y_(m), z_(m)) and orientation α_(m), β_(m)of mirror 112. To calibrate 542 fisheye lens 110, a projection functionprovided by the manufacturer can be used, or any of the othercalibration methods that are well known in the art can be employed.

In an exemplary embodiment, calibration 544 of mirror 112 is performedby searching for a solution {tilde over (M)}=(x_(m), y_(m), z_(m),α_(m), β_(m)) in an R⁵ space (M) for which corresponding local featuresin the overlapping fields of view of the two views are aligned in thepanoramic space. The camera is first placed in an environment with alarge number of distant features, particularly in the overlappingregion. Nearby features should be avoided as they would have largerparallax and hence tend to introduce errors in the alignment process.The solution is initialized to the design values, {tilde over(M)}=(0,0,h_(m)+h_(l),0,0), and then, as shown in FIG. 6, amirror-panorama 610 and a fisheye-panorama 612 are generated for theoverlapping region. As shown in FIG. 6, a program, e.g., a programrunning a scale-invariant feature transform (SIFT) algorithm, is used todetermine 530 a corresponding set of features from the two panoramas610, 612. An extracted feature can be represented in sphericalcoordinates. In a pruned feature set, let the ith feature in themirror-panorama 610 be denoted by F_(m) ^((i))=[ƒ_(m) ^((i)),φ_(m)^((i))] and the corresponding feature in the fisheye-panorama 612 bedenoted by F_(ƒ) ^((i))=[θ_(ƒ) ^((i)),φ_(ƒ) ^((i))]. The problem offinding the solution {tilde over (M)} can then be setup as aminimization of an objective function that is the sum of the square ofthe errors in the alignment of the corresponding features:

$\begin{matrix}{\underset{\overset{\sim}{M}}{\arg \mspace{11mu} \min}\mspace{11mu} {\sum{\left\lbrack {F_{f}^{(i)} - {F_{m}^{(i)}(M)}} \right\rbrack^{2}.}}} & (6)\end{matrix}$

During this optimization, it is not necessarily required to search forthe solution in the entire R⁵ space. Rather a small interval ΔM centeredat the ideal design parameter values can be searched for instead. In oneembodiment, the size of the interval ΔM is chosen based on themanufacturing tolerances and the ambiguity in the position of fisheyelens viewpoint O.

Continuing with FIG. 5, in one exemplary embodiment, once the system 100has been calibrated 540 a look-up table is generated 550. Each pixel inthe panorama can be represented by a spherical coordinate/direction. Thecorresponding location for each pixel is then found 552 in the capturedimage. Referring to FIG. 2(a), each panorama pixel (θ_(p),φ_(p)) may liein one of at least three different areas: (i) it may lie in the directfield of view of fisheye lens 110 (area A_(ƒ)), (ii) it may lie in thefield of view of mirror 112 (area A_(m)), or (iii) it may lie both inthe direct field of view of fisheye lens 110 and in the field of view ofmirror 112 (overlap area A_(o)). If (θ_(p),φ_(p)) is in the field ofview of fisheye lens 110, then its corresponding location in thecaptured image is calculated 554 by

(ƒ(θ_(p))cos φ_(p),ƒ(θ_(p))sin φ_(p))  (7)

where ƒ(θ_(p)) is the fisheye projection function, discussed above.

If the pixel is in the field of view of mirror 112, then itscorresponding location in the captured image is calculated 556 bysolving for the outgoing ray (θ_(c),φ_(c)) from fisheye lens 110 whichwould finally proceed in the pixel (θ_(p),φ_(p))'s direction after beingreflected by mirror 112. This is a non-liner problem and can be solvedas follows. Let R(θ,φ) be the direction in which a ray (θ,φ) from theorigin O would proceed after being reflected by mirror 112. Then, thefollowing optimization equation solves for (θ_(c),φ_(c)) in a leastsquares sense:

$\begin{matrix}{{\underset{({\theta_{c},\varphi_{c}})}{{\arg \mspace{11mu} \min}\mspace{11mu}}\left\lbrack {\left( {\theta_{p},\varphi_{p}} \right) - {R\left( {\theta_{c},\varphi_{c}} \right)}} \right\rbrack}^{2}.} & (8)\end{matrix}$

The image location can then be given by:

(ƒ(θ_(c))cos φ_(c),ƒ(θ_(c))sin φ_(c)).  (9)

Though solving the above problem for each pixel in the field of view ofmirror 112 can be computationally expensive, it is not necessary becausethe reflecting point on mirror 112 varies smoothly with the incoming raydirection, and therefore, the above can be solved for discrete intervalsand interpolated for the in-between pixels.

If the pixel lies in the overlapping region, its location in thepanorama is calculated 558 by both of the above methods and a linearfeathering technique can be used 559 to smooth the blending. The entiremapping between the panorama and the captured image is then used togenerate 550 look-up table. The look-up table is then used to map 560captured images to high resolution panoramas in real time, enabling realtime video applications. The calibration process and the mappingcalculations are performed once for any given system 100, and do notneed to be repeated unless there is some change in the systemconfiguration.

It will be understood that the foregoing is only illustrative of theprinciples described herein, and that various modifications can be madeby those skilled in the art without departing from the scope and spiritof the disclosed subject matter. For example, the system and methodsdescribed herein are used for panoramic imaging. It is understood thatthat techniques described herein are useful in connection with stillimaging and video imaging. Moreover, features of embodiments describedherein may be combined and/or rearranged to create new embodiments.

We claim:
 1. A mirror assembly adapted for use in a panoramic imagingsystem including a fisheye lens having a direct field of view and adetector, comprising: a mirror having a mirror field of view andconfigured to reflect an image of the mirror field of view through thefisheye lens; and a housing having a first end and a second end, themirror being secured proximate to the first end, and the second endhaving an engagement portion for securing the panoramic imaging systemthereto to position the mirror assembly relative to the fisheye lens andthe detector such that a first portion of the panoramic imagecorresponding to the direct field of view and a second portion of thepanoramic image corresponding to the mirror field of view are incidenton the detector, the first portion of the panoramic image having aportion overlapping the second portion of the panoramic image;
 2. Themirror assembly of claim 1, wherein the mirror comprises a sphericalmirror.
 3. The mirror assembly of claim 1, wherein the mirror comprisesa mirror having a radius of curvature and a width predetermined as afunction of a position of the mirror with respect to the fisheye lens.4. The mirror assembly of claim 1, wherein the housing further comprisesa cylindrical transparent wall disposed between the first end and thesecond end.
 5. The mirror assembly of claim 1, wherein the engagementportion comprises a screw thread.
 6. The mirror assembly of claim 1,wherein, when the engagement portion is adapted to position the mirrorcoaxially with regard to the fisheye lens.
 7. The mirror assembly ofclaim 1, wherein, when the engagement portion is adapted to position themirror assembly such that a lower angle of a panoramic field of view ofthe panoramic imaging system is less than or equal to a lower angle ofthe mirror field of view.
 8. The mirror assembly of claim 1, wherein,when the engagement portion is adapted to position the mirror assemblysuch that an angle of overlap between the direct field of view and themirror field of view is equal to the difference between a lower angle ofthe direct field of view and an upper angle of the mirror field of view.9. The mirror assembly of claim 1, further comprising: a processor foroperatively coupling to the detector; and a memory operatively coupledto the processor, the memory storing program instructions, and wherein,when the mirror assembly is secured to the panoramic imaging system, theprogram instructions being executed by the processor cause the processorto: capture the first portion of the panoramic image corresponding tothe direct field of view transmitted directly through the fisheye lens,and capture the second portion of the panoramic image corresponding tothe mirror field of view reflected from the mirror through the fisheyelens.
 10. The mirror assembly of claim 9, wherein the programinstructions being executed by the processor cause the processor to:receive the captured first and second portions of the panoramic imagefrom the detector, and combine the first and second portions of thepanoramic image to form the panoramic image.
 11. The mirror assembly ofclaim 10, wherein the combining of the first portion of the panoramicimage and the second portion of the panoramic image is performedutilizing a feature matching algorithm.
 12. The mirror assembly of claim10, wherein the program instructions being executed by the processorcause the processor to smooth the overlapping portion of the first andsecond portions of the panoramic image.
 13. The mirror assembly of claim9, wherein the memory stores program instructions that when executed bythe processor, further cause the processor to calibrate the system. 14.The mirror assembly of claim 13, wherein the calibrating the systemcomprises calibrating the mirror by determining the center of curvatureand orientation of the mirror.
 15. The mirror assembly of claim 9,wherein the program instructions being executed by the processor causethe processor to generate a table of correspondence between pixellocations on the first portion of the panoramic image and pixellocations on the second portion of the panoramic image.